{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参考链接：https://github.com/dmmiller612/bert-extractive-summarizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "抽取式摘要\n",
    "- 首先对句子编码\n",
    "- 然后聚类算法，找到最接近中心的句子\n",
    "- 然后共指解析，找到句子中代词的含义"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原始连接：https://keras.io/examples/nlp/text_extraction_with_bert/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T01:28:08.474387Z",
     "start_time": "2020-06-04T01:28:06.858138Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import json\n",
    "import string\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tokenizers import BertWordPieceTokenizer\n",
    "from transformers import BertTokenizer, TFBertModel, BertConfig\n",
    "\n",
    "max_len = 384\n",
    "configuration = BertConfig()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据预处理\n",
    "给定上下文，每个上下文对应一些问题，问题的答案为上下文中连续的几个单词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T01:28:09.544998Z",
     "start_time": "2020-06-04T01:28:09.514303Z"
    }
   },
   "outputs": [],
   "source": [
    "model_path = \"../../H/models/huggingface/tensorflow/bert-base-uncased/\"\n",
    "slow_tokenizer = BertTokenizer.from_pretrained(model_path)\n",
    "\n",
    "# 基于 PreTrainedTokenizerFast 的分词器，使用与传统分词器有差别\n",
    "tokenizer = BertWordPieceTokenizer(model_path + \"vocab.txt\", lowercase=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T01:28:15.837078Z",
     "start_time": "2020-06-04T01:28:15.455415Z"
    }
   },
   "outputs": [],
   "source": [
    "# 原始文本\n",
    "\n",
    "train_path = \"../datasets/textSum/train-v1.1.json\"\n",
    "eval_path = \"../datasets/textSum/dev-v1.1.json\"\n",
    "\n",
    "with open(train_path) as f:\n",
    "    raw_train_data = json.load(f)\n",
    "\n",
    "with open(eval_path) as f:\n",
    "    raw_eval_data = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:15:48.072631Z",
     "start_time": "2020-06-04T02:15:48.062298Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n",
       "  'qas': [{'answers': [{'answer_start': 515,\n",
       "      'text': 'Saint Bernadette Soubirous'}],\n",
       "    'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n",
       "    'id': '5733be284776f41900661182'},\n",
       "   {'answers': [{'answer_start': 188, 'text': 'a copper statue of Christ'}],\n",
       "    'question': 'What is in front of the Notre Dame Main Building?',\n",
       "    'id': '5733be284776f4190066117f'},\n",
       "   {'answers': [{'answer_start': 279, 'text': 'the Main Building'}],\n",
       "    'question': 'The Basilica of the Sacred heart at Notre Dame is beside to which structure?',\n",
       "    'id': '5733be284776f41900661180'},\n",
       "   {'answers': [{'answer_start': 381,\n",
       "      'text': 'a Marian place of prayer and reflection'}],\n",
       "    'question': 'What is the Grotto at Notre Dame?',\n",
       "    'id': '5733be284776f41900661181'},\n",
       "   {'answers': [{'answer_start': 92,\n",
       "      'text': 'a golden statue of the Virgin Mary'}],\n",
       "    'question': 'What sits on top of the Main Building at Notre Dame?',\n",
       "    'id': '5733be284776f4190066117e'}]},\n",
       " {'context': \"As at most other universities, Notre Dame's students run a number of news media outlets. The nine student-run outlets include three newspapers, both a radio and television station, and several magazines and journals. Begun as a one-page journal in September 1876, the Scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the United States. The other magazine, The Juggler, is released twice a year and focuses on student literature and artwork. The Dome yearbook is published annually. The newspapers have varying publication interests, with The Observer published daily and mainly reporting university and other news, and staffed by students from both Notre Dame and Saint Mary's College. Unlike Scholastic and The Dome, The Observer is an independent publication and does not have a faculty advisor or any editorial oversight from the University. In 1987, when some students believed that The Observer began to show a conservative bias, a liberal newspaper, Common Sense was published. Likewise, in 2003, when other students believed that the paper showed a liberal bias, the conservative paper Irish Rover went into production. Neither paper is published as often as The Observer; however, all three are distributed to all students. Finally, in Spring 2008 an undergraduate journal for political science research, Beyond Politics, made its debut.\",\n",
       "  'qas': [{'answers': [{'answer_start': 248, 'text': 'September 1876'}],\n",
       "    'question': 'When did the Scholastic Magazine of Notre dame begin publishing?',\n",
       "    'id': '5733bf84d058e614000b61be'},\n",
       "   {'answers': [{'answer_start': 441, 'text': 'twice'}],\n",
       "    'question': \"How often is Notre Dame's the Juggler published?\",\n",
       "    'id': '5733bf84d058e614000b61bf'},\n",
       "   {'answers': [{'answer_start': 598, 'text': 'The Observer'}],\n",
       "    'question': 'What is the daily student paper at Notre Dame called?',\n",
       "    'id': '5733bf84d058e614000b61c0'},\n",
       "   {'answers': [{'answer_start': 126, 'text': 'three'}],\n",
       "    'question': 'How many student news papers are found at Notre Dame?',\n",
       "    'id': '5733bf84d058e614000b61bd'},\n",
       "   {'answers': [{'answer_start': 908, 'text': '1987'}],\n",
       "    'question': 'In what year did the student paper Common Sense begin publication at Notre Dame?',\n",
       "    'id': '5733bf84d058e614000b61c1'}]},\n",
       " {'context': 'The university is the major seat of the Congregation of Holy Cross (albeit not its official headquarters, which are in Rome). Its main seminary, Moreau Seminary, is located on the campus across St. Joseph lake from the Main Building. Old College, the oldest building on campus and located near the shore of St. Mary lake, houses undergraduate seminarians. Retired priests and brothers reside in Fatima House (a former retreat center), Holy Cross House, as well as Columba Hall near the Grotto. The university through the Moreau Seminary has ties to theologian Frederick Buechner. While not Catholic, Buechner has praised writers from Notre Dame and Moreau Seminary created a Buechner Prize for Preaching.',\n",
       "  'qas': [{'answers': [{'answer_start': 119, 'text': 'Rome'}],\n",
       "    'question': 'Where is the headquarters of the Congregation of the Holy Cross?',\n",
       "    'id': '5733bed24776f41900661188'},\n",
       "   {'answers': [{'answer_start': 145, 'text': 'Moreau Seminary'}],\n",
       "    'question': 'What is the primary seminary of the Congregation of the Holy Cross?',\n",
       "    'id': '5733bed24776f41900661189'},\n",
       "   {'answers': [{'answer_start': 234, 'text': 'Old College'}],\n",
       "    'question': 'What is the oldest structure at Notre Dame?',\n",
       "    'id': '5733bed24776f4190066118a'},\n",
       "   {'answers': [{'answer_start': 356, 'text': 'Retired priests and brothers'}],\n",
       "    'question': 'What individuals live at Fatima House at Notre Dame?',\n",
       "    'id': '5733bed24776f4190066118b'},\n",
       "   {'answers': [{'answer_start': 675, 'text': 'Buechner Prize for Preaching'}],\n",
       "    'question': 'Which prize did Frederick Buechner create?',\n",
       "    'id': '5733bed24776f4190066118c'}]},\n",
       " {'context': 'The College of Engineering was established in 1920, however, early courses in civil and mechanical engineering were a part of the College of Science since the 1870s. Today the college, housed in the Fitzpatrick, Cushing, and Stinson-Remick Halls of Engineering, includes five departments of study – aerospace and mechanical engineering, chemical and biomolecular engineering, civil engineering and geological sciences, computer science and engineering, and electrical engineering – with eight B.S. degrees offered. Additionally, the college offers five-year dual degree programs with the Colleges of Arts and Letters and of Business awarding additional B.A. and Master of Business Administration (MBA) degrees, respectively.',\n",
       "  'qas': [{'answers': [{'answer_start': 487, 'text': 'eight'}],\n",
       "    'question': 'How many BS level degrees are offered in the College of Engineering at Notre Dame?',\n",
       "    'id': '5733a6424776f41900660f51'},\n",
       "   {'answers': [{'answer_start': 46, 'text': '1920'}],\n",
       "    'question': 'In what year was the College of Engineering at Notre Dame formed?',\n",
       "    'id': '5733a6424776f41900660f4e'},\n",
       "   {'answers': [{'answer_start': 126, 'text': 'the College of Science'}],\n",
       "    'question': 'Before the creation of the College of Engineering similar studies were carried out at which Notre Dame college?',\n",
       "    'id': '5733a6424776f41900660f4f'},\n",
       "   {'answers': [{'answer_start': 271, 'text': 'five'}],\n",
       "    'question': 'How many departments are within the Stinson-Remick Hall of Engineering?',\n",
       "    'id': '5733a6424776f41900660f50'},\n",
       "   {'answers': [{'answer_start': 155, 'text': 'the 1870s'}],\n",
       "    'question': 'The College of Science began to offer civil engineering courses beginning at what time at Notre Dame?',\n",
       "    'id': '5733a6424776f41900660f52'}]},\n",
       " {'context': \"All of Notre Dame's undergraduate students are a part of one of the five undergraduate colleges at the school or are in the First Year of Studies program. The First Year of Studies program was established in 1962 to guide incoming freshmen in their first year at the school before they have declared a major. Each student is given an academic advisor from the program who helps them to choose classes that give them exposure to any major in which they are interested. The program also includes a Learning Resource Center which provides time management, collaborative learning, and subject tutoring. This program has been recognized previously, by U.S. News & World Report, as outstanding.\",\n",
       "  'qas': [{'answers': [{'answer_start': 496,\n",
       "      'text': 'Learning Resource Center'}],\n",
       "    'question': 'What entity provides help with the management of time for new students at Notre Dame?',\n",
       "    'id': '5733a70c4776f41900660f64'},\n",
       "   {'answers': [{'answer_start': 68, 'text': 'five'}],\n",
       "    'question': 'How many colleges for undergraduates are at Notre Dame?',\n",
       "    'id': '5733a70c4776f41900660f62'},\n",
       "   {'answers': [{'answer_start': 155,\n",
       "      'text': 'The First Year of Studies program'}],\n",
       "    'question': 'What was created at Notre Dame in 1962 to assist first year students?',\n",
       "    'id': '5733a70c4776f41900660f63'},\n",
       "   {'answers': [{'answer_start': 647, 'text': 'U.S. News & World Report'}],\n",
       "    'question': 'Which organization declared the First Year of Studies program at Notre Dame \"outstanding?\"',\n",
       "    'id': '5733a70c4776f41900660f65'}]},\n",
       " {'context': 'The university first offered graduate degrees, in the form of a Master of Arts (MA), in the 1854–1855 academic year. The program expanded to include Master of Laws (LL.M.) and Master of Civil Engineering in its early stages of growth, before a formal graduate school education was developed with a thesis not required to receive the degrees. This changed in 1924 with formal requirements developed for graduate degrees, including offering Doctorate (PhD) degrees. Today each of the five colleges offer graduate education. Most of the departments from the College of Arts and Letters offer PhD programs, while a professional Master of Divinity (M.Div.) program also exists. All of the departments in the College of Science offer PhD programs, except for the Department of Pre-Professional Studies. The School of Architecture offers a Master of Architecture, while each of the departments of the College of Engineering offer PhD programs. The College of Business offers multiple professional programs including MBA and Master of Science in Accountancy programs. It also operates facilities in Chicago and Cincinnati for its executive MBA program. Additionally, the Alliance for Catholic Education program offers a Master of Education program where students study at the university during the summer and teach in Catholic elementary schools, middle schools, and high schools across the Southern United States for two school years.',\n",
       "  'qas': [{'answers': [{'answer_start': 358, 'text': '1924'}],\n",
       "    'question': 'The granting of Doctorate degrees first occurred in what year at Notre Dame?',\n",
       "    'id': '5733a7bd4776f41900660f6b'},\n",
       "   {'answers': [{'answer_start': 624, 'text': 'Master of Divinity'}],\n",
       "    'question': 'What type of degree is an M.Div.?',\n",
       "    'id': '5733a7bd4776f41900660f6c'},\n",
       "   {'answers': [{'answer_start': 1163,\n",
       "      'text': 'Alliance for Catholic Education'}],\n",
       "    'question': 'Which program at Notre Dame offers a Master of Education degree?',\n",
       "    'id': '5733a7bd4776f41900660f6e'},\n",
       "   {'answers': [{'answer_start': 92, 'text': '1854'}],\n",
       "    'question': 'In what year was a Master of Arts course first offered at Notre Dame?',\n",
       "    'id': '5733a7bd4776f41900660f6a'},\n",
       "   {'answers': [{'answer_start': 757,\n",
       "      'text': 'Department of Pre-Professional Studies'}],\n",
       "    'question': 'Which department at Notre Dame is the only one to not offer a PhD program?',\n",
       "    'id': '5733a7bd4776f41900660f6d'}]},\n",
       " {'context': \"The Joan B. Kroc Institute for International Peace Studies at the University of Notre Dame is dedicated to research, education and outreach on the causes of violent conflict and the conditions for sustainable peace. It offers PhD, Master's, and undergraduate degrees in peace studies. It was founded in 1986 through the donations of Joan B. Kroc, the widow of McDonald's owner Ray Kroc. The institute was inspired by the vision of the Rev. Theodore M. Hesburgh CSC, President Emeritus of the University of Notre Dame. The institute has contributed to international policy discussions about peace building practices.\",\n",
       "  'qas': [{'answers': [{'answer_start': 4,\n",
       "      'text': 'Joan B. Kroc Institute for International Peace Studies'}],\n",
       "    'question': 'What institute at Notre Dame studies  the reasons for violent conflict?',\n",
       "    'id': '5733ac31d058e614000b5ff3'},\n",
       "   {'answers': [{'answer_start': 466,\n",
       "      'text': 'President Emeritus of the University of Notre Dame'}],\n",
       "    'question': \"What is the title of Notre Dame's Theodore Hesburgh?\",\n",
       "    'id': '5733ac31d058e614000b5ff6'},\n",
       "   {'answers': [{'answer_start': 303, 'text': '1986'}],\n",
       "    'question': 'In what year was the Joan B. Kroc Institute for International Peace Studies founded?',\n",
       "    'id': '5733ac31d058e614000b5ff4'},\n",
       "   {'answers': [{'answer_start': 377, 'text': 'Ray Kroc'}],\n",
       "    'question': 'To whom was John B. Kroc married?',\n",
       "    'id': '5733ac31d058e614000b5ff5'},\n",
       "   {'answers': [{'answer_start': 360, 'text': \"McDonald's\"}],\n",
       "    'question': 'What company did Ray Kroc own?',\n",
       "    'id': '5733ac31d058e614000b5ff7'}]},\n",
       " {'context': 'The library system of the university is divided between the main library and each of the colleges and schools. The main building is the 14-story Theodore M. Hesburgh Library, completed in 1963, which is the third building to house the main collection of books. The front of the library is adorned with the Word of Life mural designed by artist Millard Sheets. This mural is popularly known as \"Touchdown Jesus\" because of its proximity to Notre Dame Stadium and Jesus\\' arms appearing to make the signal for a touchdown.',\n",
       "  'qas': [{'answers': [{'answer_start': 136, 'text': '14'}],\n",
       "    'question': 'How many stories tall is the main library at Notre Dame?',\n",
       "    'id': '5733ad384776f41900660fec'},\n",
       "   {'answers': [{'answer_start': 145, 'text': 'Theodore M. Hesburgh Library'}],\n",
       "    'question': 'What is the name of the main library at Notre Dame?',\n",
       "    'id': '5733ad384776f41900660fed'},\n",
       "   {'answers': [{'answer_start': 188, 'text': '1963'}],\n",
       "    'question': 'In what year was the Theodore M. Hesburgh Library at Notre Dame finished?',\n",
       "    'id': '5733ad384776f41900660fee'},\n",
       "   {'answers': [{'answer_start': 344, 'text': 'Millard Sheets'}],\n",
       "    'question': 'Which artist created the mural on the Theodore M. Hesburgh Library?',\n",
       "    'id': '5733ad384776f41900660fef'},\n",
       "   {'answers': [{'answer_start': 394, 'text': 'Touchdown Jesus'}],\n",
       "    'question': 'What is a common name to reference the mural created by Millard Sheets at Notre Dame?',\n",
       "    'id': '5733ad384776f41900660ff0'}]},\n",
       " {'context': 'Notre Dame is known for its competitive admissions, with the incoming class enrolling in fall 2015 admitting 3,577 from a pool of 18,156 (19.7%). The academic profile of the enrolled class continues to rate among the top 10 to 15 in the nation for national research universities. The university practices a non-restrictive early action policy that allows admitted students to consider admission to Notre Dame as well as any other colleges to which they were accepted. 1,400 of the 3,577 (39.1%) were admitted under the early action plan. Admitted students came from 1,311 high schools and the average student traveled more than 750 miles to Notre Dame, making it arguably the most representative university in the United States. While all entering students begin in the College of the First Year of Studies, 25% have indicated they plan to study in the liberal arts or social sciences, 24% in engineering, 24% in business, 24% in science, and 3% in architecture.',\n",
       "  'qas': [{'answers': [{'answer_start': 109, 'text': '3,577'}],\n",
       "    'question': 'How many incoming students did Notre Dame admit in fall 2015?',\n",
       "    'id': '5733ae924776f41900661014'},\n",
       "   {'answers': [{'answer_start': 138, 'text': '19.7%'}],\n",
       "    'question': 'What percentage of students were admitted to Notre Dame in fall 2015?',\n",
       "    'id': '5733ae924776f41900661013'},\n",
       "   {'answers': [{'answer_start': 213,\n",
       "      'text': 'the top 10 to 15 in the nation'}],\n",
       "    'question': 'Where does Notre Dame rank in terms of academic profile among research universities in the US?',\n",
       "    'id': '5733ae924776f41900661015'},\n",
       "   {'answers': [{'answer_start': 488, 'text': '39.1%'}],\n",
       "    'question': 'What percentage of students at Notre Dame participated in the Early Action program?',\n",
       "    'id': '5733ae924776f41900661016'},\n",
       "   {'answers': [{'answer_start': 618, 'text': 'more than 750 miles'}],\n",
       "    'question': 'How many miles does the average student at Notre Dame travel to study there?',\n",
       "    'id': '5733ae924776f41900661017'}]},\n",
       " {'context': 'In 2015-2016, Notre Dame ranked 18th overall among \"national universities\" in the United States in U.S. News & World Report\\'s Best Colleges 2016. In 2014, USA Today ranked Notre Dame 10th overall for American universities based on data from College Factual. Forbes.com\\'s America\\'s Best Colleges ranks Notre Dame 13th among colleges in the United States in 2015, 8th among Research Universities, and 1st in the Midwest. U.S. News & World Report also lists Notre Dame Law School as 22nd overall. BusinessWeek ranks Mendoza College of Business undergraduate school as 1st overall. It ranks the MBA program as 20th overall. The Philosophical Gourmet Report ranks Notre Dame\\'s graduate philosophy program as 15th nationally, while ARCHITECT Magazine ranked the undergraduate architecture program as 12th nationally. Additionally, the study abroad program ranks sixth in highest participation percentage in the nation, with 57.6% of students choosing to study abroad in 17 countries. According to payscale.com, undergraduate alumni of University of Notre Dame have a mid-career median salary $110,000, making it the 24th highest among colleges and universities in the United States. The median starting salary of $55,300 ranked 58th in the same peer group.',\n",
       "  'qas': [{'answers': [{'answer_start': 32, 'text': '18th overall'}],\n",
       "    'question': 'Where did U.S. News & World Report rank Notre Dame in its 2015-2016 university rankings?',\n",
       "    'id': '5733afd3d058e614000b6045'},\n",
       "   {'answers': [{'answer_start': 362, 'text': '8th'}],\n",
       "    'question': 'Forbes.com placed Notre Dame at what position compared to other US research universities?',\n",
       "    'id': '5733afd3d058e614000b6047'},\n",
       "   {'answers': [{'answer_start': 565, 'text': '1st overall'}],\n",
       "    'question': 'The undergrad school at the Mendoza College of Business was ranked where according to BusinessWeek?',\n",
       "    'id': '5733afd3d058e614000b6048'},\n",
       "   {'answers': [{'answer_start': 155, 'text': 'USA Today'}],\n",
       "    'question': 'In 2014 what entity named Notre Dame 10th best of all American universities?',\n",
       "    'id': '5733afd3d058e614000b6046'},\n",
       "   {'answers': [{'answer_start': 918, 'text': '57.6%'}],\n",
       "    'question': 'What percentage of Notre Dame students decide to study abroad?',\n",
       "    'id': '5733afd3d058e614000b6049'}]}]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 数据样本:一个 文本段落 对应多个 问题-答案 对\n",
    "raw_train_data['data'][0]['paragraphs'][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:33:03.201749Z",
     "start_time": "2020-06-04T02:33:03.195672Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0),\n",
       " (0, 2),\n",
       " (3, 7),\n",
       " (7, 8),\n",
       " (8, 12),\n",
       " (12, 13),\n",
       " (14, 19),\n",
       " (20, 24),\n",
       " (25, 31),\n",
       " (32, 36),\n",
       " (37, 44),\n",
       " (45, 50),\n",
       " (51, 52),\n",
       " (52, 60),\n",
       " (61, 73),\n",
       " (73, 74),\n",
       " (75, 77),\n",
       " (78, 81),\n",
       " (82, 88),\n",
       " (89, 95),\n",
       " (96, 98),\n",
       " (99, 100),\n",
       " (100, 101),\n",
       " (101, 102),\n",
       " (102, 103),\n",
       " (0, 0)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context = 'In 2015-2016, Notre Dame ranked 18th overall among \"national universities\" in the United States in U.S.'\n",
    "tokenized_context = tokenizer.encode(context)\n",
    "\n",
    "# tokenized_context.offsets 元组列表，标识每个标记的起始索引点\n",
    "# 基于 Rust 继承自 PreTrainedTokenizerFast 的分词器才有该属性，Python 分词器无\n",
    "\n",
    "tokenized_context.offsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:46:23.968272Z",
     "start_time": "2020-06-04T02:46:23.962176Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101--> [CLS]\n",
      "1999--> in\n",
      "2054--> what\n",
      "2149--> us\n",
      "2110--> state\n",
      "2106--> did\n",
      "28045--> kathmandu\n",
      "2034--> first\n",
      "5323--> establish\n",
      "2019--> an\n",
      "2248--> international\n",
      "3276--> relationship\n",
      "1029--> ?\n",
      "102--> [SEP]\n"
     ]
    }
   ],
   "source": [
    "question = 'In what US state did Kathmandu first establish an international relationship?'\n",
    "question_ids = tokenizer.encode(question)\n",
    "for idx in question_ids.ids:\n",
    "    print(f\"{idx}-->\", tokenizer.id_to_token(idx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:53:44.613478Z",
     "start_time": "2020-06-04T02:53:44.603340Z"
    }
   },
   "outputs": [],
   "source": [
    "class SquadExample:\n",
    "    def __init__(self, question, context, start_char_idx, answer_text, all_answers):\n",
    "        self.question = question\n",
    "        self.context = context\n",
    "        self.start_char_idx = start_char_idx\n",
    "        self.answer_text = answer_text\n",
    "        self.all_answers = all_answers\n",
    "        self.skip = False\n",
    "\n",
    "    def preprocess(self):\n",
    "        # 从完整的文本中抽取连续的标记，作为问题的答案\n",
    "        # 完整的文本\n",
    "        context = self.context\n",
    "        \n",
    "        # 问题，该问题的答案在完整的文本中\n",
    "        question = self.question\n",
    "        \n",
    "        # 问题的答案\n",
    "        answer_text = self.answer_text\n",
    "        \n",
    "        # 答案位于完整文本中的 起点\n",
    "        start_char_idx = self.start_char_idx\n",
    "\n",
    "        # 删除多余的空格\n",
    "        context = \" \".join(str(context).split())\n",
    "        question = \" \".join(str(question).split())\n",
    "        answer = \" \".join(str(answer_text).split())\n",
    "\n",
    "        # 答案位于完整文本中的 终点\n",
    "        end_char_idx = start_char_idx + len(answer)\n",
    "        if end_char_idx >= len(context):\n",
    "            self.skip = True\n",
    "            return\n",
    "\n",
    "        # 标记完整文本中哪些 **字符** 是 答案\n",
    "        is_char_in_ans = [0] * len(context)\n",
    "        for idx in range(start_char_idx, end_char_idx):\n",
    "            is_char_in_ans[idx] = 1\n",
    "\n",
    "        # 完整文本向量化，单词会拆分成子词\n",
    "        tokenized_context = tokenizer.encode(context)\n",
    "\n",
    "        # tokenized_context.offsets 元组列表，标识每个单词的起始索引点\n",
    "        # 标记完整文本中哪些 **单词** 属于答案\n",
    "        ans_token_idx = []\n",
    "        for idx, (start, end) in enumerate(tokenized_context.offsets):\n",
    "            if sum(is_char_in_ans[start:end]) > 0:\n",
    "                ans_token_idx.append(idx)\n",
    "\n",
    "        if len(ans_token_idx) == 0:\n",
    "            self.skip = True\n",
    "            return\n",
    "\n",
    "        # 答案中的 **子词** 在完整文本中的起始索引点\n",
    "        start_token_idx = ans_token_idx[0]\n",
    "        end_token_idx = ans_token_idx[-1]\n",
    "\n",
    "        # 问题向量化,自动添加了 [CLS] 和 [SEP] 标签了\n",
    "        tokenized_question = tokenizer.encode(question)\n",
    "\n",
    "        # 模型输入，将 完整文本 和 问题 拼接起来\n",
    "        input_ids = tokenized_context.ids + tokenized_question.ids[1:]\n",
    "        token_type_ids = [0] * len(tokenized_context.ids) + [1] * len(\n",
    "            tokenized_question.ids[1:]\n",
    "        )\n",
    "        attention_mask = [1] * len(input_ids)\n",
    "\n",
    "        # 掩码，表征填充\n",
    "        # 如果 完整文本+问题 超长，则丢弃\n",
    "        padding_length = max_len - len(input_ids)\n",
    "        if padding_length > 0:  # pad\n",
    "            input_ids = input_ids + ([0] * padding_length)\n",
    "            attention_mask = attention_mask + ([0] * padding_length)\n",
    "            token_type_ids = token_type_ids + ([0] * padding_length)\n",
    "        elif padding_length < 0:  # skip\n",
    "            self.skip = True\n",
    "            return\n",
    "\n",
    "        # 模型输入数据:\n",
    "        # 完整文本+问题 的向量\n",
    "        self.input_ids = input_ids\n",
    "        self.token_type_ids = token_type_ids\n",
    "        self.attention_mask = attention_mask\n",
    "        self.start_token_idx = start_token_idx\n",
    "        self.end_token_idx = end_token_idx\n",
    "        self.context_token_to_char = tokenized_context.offsets\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:56:12.065833Z",
     "start_time": "2020-06-04T02:56:12.059259Z"
    }
   },
   "outputs": [],
   "source": [
    "# 从原始语料中提取需要的文本\n",
    "def create_squad_examples(raw_data):\n",
    "    squad_examples = []\n",
    "    for item in raw_data[\"data\"]:\n",
    "        for para in item[\"paragraphs\"]:\n",
    "            context = para[\"context\"]\n",
    "            for qa in para[\"qas\"]:\n",
    "                # 一个问题\n",
    "                question = qa[\"question\"]\n",
    "\n",
    "                # 可能对应多个答案\n",
    "                answer_text = qa[\"answers\"][0][\"text\"]\n",
    "                all_answers = [_[\"text\"] for _ in qa[\"answers\"]]\n",
    "\n",
    "                # 第一个答案的起始索引点\n",
    "                start_char_idx = qa[\"answers\"][0][\"answer_start\"]\n",
    "                squad_eg = SquadExample(\n",
    "                    question,\n",
    "                    context,\n",
    "                    start_char_idx,\n",
    "                    answer_text,\n",
    "                    all_answers,\n",
    "                )\n",
    "                squad_eg.preprocess()\n",
    "                squad_examples.append(squad_eg)\n",
    "    return squad_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:56:40.138975Z",
     "start_time": "2020-06-04T02:56:12.732635Z"
    }
   },
   "outputs": [],
   "source": [
    "train_squad_example = create_squad_examples(raw_train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:56:40.657722Z",
     "start_time": "2020-06-04T02:56:40.655594Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "87599"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_squad_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:56:48.516073Z",
     "start_time": "2020-06-04T02:56:48.509285Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_inputs_targets(squad_examples):\n",
    "    dataset_dict = {\n",
    "        \"input_ids\": [],\n",
    "        \"token_type_ids\": [],\n",
    "        \"attention_mask\": [],\n",
    "        \"start_token_idx\": [],\n",
    "        \"end_token_idx\": [],\n",
    "    }\n",
    "    for item in squad_examples:\n",
    "        if item.skip == False:\n",
    "            for key in dataset_dict:\n",
    "                dataset_dict[key].append(getattr(item, key))\n",
    "    for key in dataset_dict:\n",
    "        dataset_dict[key] = np.array(dataset_dict[key])\n",
    "\n",
    "    x = [\n",
    "        dataset_dict[\"input_ids\"],\n",
    "        dataset_dict[\"token_type_ids\"],\n",
    "        dataset_dict[\"attention_mask\"],\n",
    "    ]\n",
    "\n",
    "    # 目标数据为：起始点和终点索引值，两点之间的文本即为答案\n",
    "    y = [dataset_dict[\"start_token_idx\"], dataset_dict[\"end_token_idx\"]]\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:56:56.261647Z",
     "start_time": "2020-06-04T02:56:52.502905Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "87599 training points created\n"
     ]
    }
   ],
   "source": [
    "x_train, y_train = create_inputs_targets(train_squad_example)\n",
    "print(f\"{len(train_squad_example)} training points created\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:57:07.189018Z",
     "start_time": "2020-06-04T02:57:02.538877Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10570 evaluation points created.\n"
     ]
    }
   ],
   "source": [
    "eval_squad_examples = create_squad_examples(raw_eval_data)\n",
    "x_eval, y_eval = create_inputs_targets(eval_squad_examples)\n",
    "print(f\"{len(eval_squad_examples)} evaluation points created.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:57:16.064247Z",
     "start_time": "2020-06-04T02:57:16.056091Z"
    }
   },
   "outputs": [],
   "source": [
    "def create_model():\n",
    "    ## BERT encoder\n",
    "    encoder = TFBertModel.from_pretrained(model_path)\n",
    "\n",
    "    ## QA Model\n",
    "    input_ids = layers.Input(shape=(max_len, ), dtype=tf.int32)\n",
    "    token_type_ids = layers.Input(shape=(max_len, ), dtype=tf.int32)\n",
    "    attention_mask = layers.Input(shape=(max_len, ), dtype=tf.int32)\n",
    "    embedding = encoder(input_ids,\n",
    "                        token_type_ids=token_type_ids,\n",
    "                        attention_mask=attention_mask)[0]\n",
    "\n",
    "    start_logits = layers.Dense(1, name=\"start_logit\",\n",
    "                                use_bias=False)(embedding)\n",
    "    start_logits = layers.Flatten()(start_logits)\n",
    "\n",
    "    end_logits = layers.Dense(1, name=\"end_logit\", use_bias=False)(embedding)\n",
    "    end_logits = layers.Flatten()(end_logits)\n",
    "\n",
    "    start_probs = layers.Activation(keras.activations.softmax)(start_logits)\n",
    "    end_probs = layers.Activation(keras.activations.softmax)(end_logits)\n",
    "\n",
    "    model = keras.Model(\n",
    "        inputs=[input_ids, token_type_ids, attention_mask],\n",
    "        outputs=[start_probs, end_probs],\n",
    "    )\n",
    "    loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
    "    optimizer = keras.optimizers.Adam(lr=5e-5)\n",
    "    model.compile(optimizer=optimizer, loss=[loss, loss])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:57:40.428311Z",
     "start_time": "2020-06-04T02:57:35.408460Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 384)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_3 (InputLayer)            [(None, 384)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, 384)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "tf_bert_model (TFBertModel)     ((None, 384, 768), ( 109482240   input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "start_logit (Dense)             (None, 384, 1)       768         tf_bert_model[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "end_logit (Dense)               (None, 384, 1)       768         tf_bert_model[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "flatten (Flatten)               (None, 384)          0           start_logit[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 384)          0           end_logit[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_7 (Activation)       (None, 384)          0           flatten[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_8 (Activation)       (None, 384)          0           flatten_1[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 109,483,776\n",
      "Trainable params: 109,483,776\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = create_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 回调函数，计算验证性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-04T02:57:40.853234Z",
     "start_time": "2020-06-04T02:57:40.850411Z"
    }
   },
   "outputs": [],
   "source": [
    "def normalize_text(text):\n",
    "    text = text.lower()\n",
    "\n",
    "    # 删除标点符号\n",
    "    exclude = set(string.punctuation)\n",
    "    text = \"\".join(ch for ch in text if ch not in exclude)\n",
    "\n",
    "    # 删除冠词\n",
    "    regex = re.compile(r\"\\b(a|an|the)\\b\", re.UNICODE)\n",
    "    text = re.sub(regex, \" \", text)\n",
    "\n",
    "\n",
    "    # 删除多余的空格\n",
    "    text = \" \".join(text.split())\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-03T11:26:06.827541Z",
     "start_time": "2020-06-03T11:26:06.819657Z"
    }
   },
   "outputs": [],
   "source": [
    "class ExactMatch(keras.callbacks.Callback):\n",
    "    \"\"\"\n",
    "    Each `SquadExample` object contains the character level offsets for each token\n",
    "    in its input paragraph. We use them to get back the span of text corresponding\n",
    "    to the tokens between our predicted start and end tokens.\n",
    "    All the ground-truth answers are also present in each `SquadExample` object.\n",
    "    We calculate the percentage of data points where the span of text obtained\n",
    "    from model predictions matches one of the ground-truth answers.\n",
    "    \"\"\"\n",
    "    def __init__(self, x_eval, y_eval):\n",
    "        self.x_eval = x_eval\n",
    "        self.y_eval = y_eval\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        pred_start, pred_end = self.model.predict(self.x_eval)\n",
    "        count = 0\n",
    "        eval_examples_no_skip = [\n",
    "            _ for _ in eval_squad_examples if _.skip == False\n",
    "        ]\n",
    "        for idx, (start, end) in enumerate(zip(pred_start, pred_end)):\n",
    "            squad_eg = eval_examples_no_skip[idx]\n",
    "            offsets = squad_eg.context_token_to_char\n",
    "            start = np.argmax(start)\n",
    "            end = np.argmax(end)\n",
    "            if start >= len(offsets):\n",
    "                continue\n",
    "            pred_char_start = offsets[start][0]\n",
    "            if end < len(offsets):\n",
    "                pred_char_end = offsets[end][1]\n",
    "                pred_ans = squad_eg.context[pred_char_start:pred_char_end]\n",
    "            else:\n",
    "                pred_ans = squad_eg.context[pred_char_start:]\n",
    "\n",
    "            normalized_pred_ans = normalize_text(pred_ans)\n",
    "            normalized_true_ans = [\n",
    "                normalize_text(_) for _ in squad_eg.all_answers\n",
    "            ]\n",
    "            if normalized_pred_ans in normalized_true_ans:\n",
    "                count += 1\n",
    "        acc = count / len(self.y_eval[0])\n",
    "        print(f\"\\nepoch={epoch+1}, exact match score={acc:.2f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-03T12:09:18.300538Z",
     "start_time": "2020-06-03T11:39:17.806863Z"
    }
   },
   "outputs": [],
   "source": [
    "exact_match_callback = ExactMatch(x_eval, y_eval)\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    epochs=1,\n",
    "    verbose=2,\n",
    "    batch_size=64,  # 超出内存，减小批的大小\n",
    "    callbacks=[exact_match_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 输入是文本段落和问题，拼接起来，max_len=384，每个标记对应的词向量又是 768，所以最终超内存！！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "775px",
    "left": "986px",
    "right": "20px",
    "top": "129px",
    "width": "791px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
